Speaker verification generally relates to verifying the identity of a person based on characteristics of the person's voice. Some computing devices allow a user to “enroll” with the device by providing to the device one or more samples of speech spoken by the user, from which a speaker model representing the user's voice is determined. Subsequent speech samples received at the device may then be processed and evaluated with respect to the speaker model to verify a user's identity.